Conditional Recurrent Neural Networks

Aim of the Project

• Problem Statement: The goal is to find a novel chemical entities that will bind to a receptor using a generative deep-learning model.

• BUT there are about 10^60 possible clinical candidates! (< 500 Da and meet Lapinski’s rule)

• Traditional approach: Search for properties via databases and test.

• Generative D-L approach: Train on known entities and their properties, and predict which ones might work

The Target: ACKR3 – a GPC-receptor

• Motivation: ACKR3 is plays a role in:
  • angiogenesis (development of new blood vessels);
    
  • tumor growth and metastatic processes;
  • neurological conditions; etc.

Source (Yen 2022)

Related: Generative Models

• Bjerrum (2017) and Gupta (2017): Shows the appeoach

• Marwin H. S. Segler and Thierry Kogej (2018) explored the use of the RNN to create de novo drugs for finding compounds with an inhibitory concentration for Plasmodium falciparum.

• Ligand-binding: Our work builds upon from existing DL methods using ligand binding as objective like in Stepniewska (2018) that used a CNN.

RNN Extensions:

• Olivecrona (2017): gradient clipping

• Preuer (2018): 2 CNN layers pre-pended

• Kotsias (2020): Conditional with pre-pended QSAR properties

• Xu (2021): Conditional Coulomb matrix of the pocket

–> I use a conditional RNN to generate the molecules

Data: Sources

• ACKR3 ligands:
  • Vreije Universiteit NL (Riemens 2023);
  • InterAx; and,
  • papers on ACKR3.
• ZINC dataset: Known clinical molecules 10’000s of entities
• ChEMBL: Ligands known to work with GPCR protein receptors

Data: Aggregation

KNIME Workflow

Data: Relevant Features (RDKit/gnina)

Data

Distribution of mass (Da) amongst different classes of molecules

Data: Mild Correlation between Binding and pCHEMBL

KNIME reports a modest but statistcaly significant correlation of 0.236 between gnina docking score the pChemBL (p-value=0.000 on 575 degrees of freedom).

Example of Inverse Agonist

Example of ACT-1004, an inverse agonist

Model: Trials and Tribulations

• Iterative Process:

  • VAE graph info sparse encoding into network and features error prone

  • GAN unstable, hardware issue optimizing

  • –> The RNN seemed to tune the best out of the box, and give relatively similar SMILEs without any tuning.

Model Architecture: RNN simple, yet powerful

• Chemistry can be fairly well expressed in terms of a sequence of elements and radicals

• Given the nature of the SMILE text small storage and memory requirements

• Simple logic, works on minor hardware (MacPro w/ nVidia GPU)

• Clinical properties integrated with conditional RNN architecture

Model: Schema

Model: Conditional RNN Formulation

Both the output and hidden process condition on the chemical properties of the molecule.

• Hidden process \[ h_t = f(W_{xh}x_t + W_{hh} h_{t-1} +\mathbf{W}_{ch}\mathbf{c}+b_h) \\ \]

• Output process \[ y_t = g(W_{hy} h_t +\mathbf{W}_{cy}\mathbf{c}+b_y) \]

Model: Core Model Formulation

Model: Loss

• Batch size restricted due to hardware limitations
• Optimizer was Adam with a default learning rate of: 0.0001
• Loss function was sparse cross-entropy, which matches the encoding of each character as an integer: \[ L = - \frac{1}{N} \Sigma_{i=1}^T [ln(p_{y=c,i})] \] —

Models: Overview of RNN Models

Results: Accuracy

• Very low out of sample validity, lots of invalid SMILES (97.5%) depending
• Gains low after 64+ epochs

Results: Validity

Results: DeNovo Shifting Distribution Higher

Results: Example DeNovo Molecules

Results: de Novo like Training Distribution

t-SNE

Results: A (un)known SMILE

• String: “Cc1cc(CN)n(n1)[C@H]1CC@@Hc1cc2cc(C)ccc2n1Cc1ccc(Cl)cc1”
• Received a gnina CNN ligand-ACKR3 binding score of 0.961

Nicotinamide N-methyltransferase (NNMT)

Boltz-2 Simulation: Baseline

The inverse agonist ACT-1004-1239

Boltz-2 Simulation: Potential Candidate

Candidate inverse agonist A

Boltz-2 Simulation: Bad Candidate

Candidate D penetrates the trans-membrane helix: both “bad” docking, and the molecule. We can see a long-chain polymer (orange) penetrates the trans-membrane helices. This candidate is likely spurious or will be disruptive to the GPCR.

Discussion: Challenges

• Low-level hardware issues (galore!): gru activation function, GPU libraries, etc.
• Compute resources scarce: Binding score has taken 7+ months of compute time to generate! Thousands of files, slow cycle times.
• Large IT setup and pre-work: Python versionning and environment, undocumented tools
• Validating the issue encountered is the de novo ligand.

Discussion: Future Directions & Ideas

• Add topological information of the molecule through graph or structural fingerprints
• A larger datasets with all GPCR ligands would be useful including those that have been docked in a complex.
• An interesting idea would be to investigate to include simulation input as a CNN.

Conclusion

• Achievements:
  • Scored ~97’000 known candidate ligands for their potential to dock in complex with ACKR3 using gnina’s CNN docking function
  • Used binding score to find ~200 novel candidate ligands for ACKR3 obtaining some therapeutic compound leads
• Key Takeaway: Demonstrates the potential of conditional RNNs for developing candidate molecules tailored to a specific domain.

Thank You & Questions?

• Thanks to Tomek at InterAx Biotech for showing me MD and helping me with the the biology
• Thanks Manuel Dömer for putting up with my inner conflict
• Ready for questions.

Appendix: F-Tree Similarity Search

F-Tree similarity search on VUF-0016840 using infiniSee from BioSolveIT finds many similar matches on a huge library (billions). Compounds may be too similar to the target vs. generative techniques.

Appendix: Feature Summary Statistics

Appendix: Feature Summary Statistics

Appendix: Feature Summary Statistics

Appendix: Feature Summary Statistics

Appendix: Improvement of the RNN for SMILE chemistry

We played around with replacement of digraphs, trigraphs with singletons, like.

• “As”: “🜺”, # Alchemical Arsenic
• “Ag”: “☽”, # Alchemical symbol for Silver (Moon)
• “Na”: “钠”, # Nu for Sodium whose Mandarin name is “nà”
• “Be”: “铍”, # Chinese character for Beryllium
• “Bi”: “♆”, # Alchemical Bisthmuth
• …

Gains were small to nil. As the combinations lengthen, there are fewer of them to replace.